Azure Machine Learning supports data drift monitoring
through the use of datasets. You can capture new feature
data in a dataset and compare it to the dataset with which the
model was trained. It's common for
organizations to continue to collect new data after a
model has been trained. For example, a health clinic might use diagnostic
measurements from previous patients to train a model that predicts
diabetes likelihood, but continued to collect the same diagnostic measurements
from all new patients. The clinic's data scientists could then periodically compare the growing collection
of new data to the original training data, and identify any
changing data trends that might affect
model accuracy. To monitor data drift
using registered datasets, you need to register two
datasets: A baseline dataset, which is usually the
original training data, and a target dataset
that will be compared to the baseline based
on time intervals. This dataset requires
a column for each feature you want to
compare and a timestamp column, so the rate of data
drift can be measured. It's important to note
that you can configure a deployed service to collect new data submitted to the
model for inferencing, which is saved in
Azure Blob Storage, and can be used as
a target dataset for data drift monitoring. See collect data from
models in production in the Azure Machine
Learning documentation for more information. You can find a link
to this resource from the additional readings at
the end of this lesson. After creating these datasets, you can define a dataset
monitor to detect data drift and trigger
alerts if the rate of drift exceeds a
specified threshold. You can create dataset
monitors using the visual interface in Azure
Machine Learning Studio, or by using the data
drift detector class in the Azure Machine
Learning SDK. After creating the
dataset monitor, you can backfill to
immediately compare the baseline dataset to existing data in
the target dataset, which backfills the
monitor based on weekly changes in data for
the previous six weeks.